Methods and apparatus for processing of a digital image

ABSTRACT

Edge information of an input digital image is obtained. Digital image data corresponding to the input digital image is transmitted to a receiver. The edge information is also transmitted to the receiver. An output digital image is obtained at the receiver by processing the digital image data received at the receiver using the edge information received at the receiver. In this way, because the edge information is transmitted separately, as side information, and does not have to be obtained at the receiver, the computational complexity requires of the receiver is low. This enables for example real-time edge-adaptive interpolation to be carried out at the receiver.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. application Ser.No. 60/891,604 filed Feb. 26, 2007, the content of which is herebyincorporated by reference in its entirety for all purposes.

FIELD OF INVENTION

The invention relates to methods and apparatus for processing of adigital image.

BACKGROUND

A number of techniques are known for processing of digital images inwhich edge information is used to improve the quality of the outputimage. For example, low resolution images may be interpolated to producehigh resolution images, with the interpolation being edge-adaptive sothat the quality of the output image, particularly around edges in theimage, is improved by removing or reducing blockiness, blurring andzigzag edges which can occur with non-edge based techniques.

A number of edge-adaptive interpolation methods are known and referencemay be made for example to “Edge-Directed Interpolation” by Jan Allebachand Ping Wah Wong, International Conference on Image Processing, 1996Proceedings, Volume 3, pages 707-710; “A New Edge-Directed ImageExpansion Scheme” by Qing Wang and Rabab Ward, 2001 InternationalConference on Image Processing, 2001 Proceedings, Volume 3, pages899-902; and “Canny Edge Based Image Expansion” by Hongjian Shi andRabab Ward, IEEE International Symposium on Circuits and Systems, 2002,Volume 1, pages I-785-I-788. In general, edge information is used tolocally adapt filter coefficients that are used during the interpolationof the input image data. Edge-adaptive interpolation methods generallyhave a high computational complexity owing to the additional edgeprocessing. Robust edge detectors require spatial filtering withmultiple filters corresponding to different edge orientations, resultingin a high computational burden.

An approach to edge-adaptive interpolation is to use edge information,which may be in the form of an explicit edge map that containsinformation about edges in the image, to guide the interpolationprocess. This is shown schematically in FIG. 1.

First, a sub-pixel edge estimator 1 obtains an accurate estimate of thehigh resolution edge map. This edge map of the interpolated (output)image can be obtained in two distinct ways. In one, a simpleinterpolation technique (such as bilinear or bicubic interpolation) canbe used to obtain an initial interpolated (higher resolution) image, andthen the estimated edge map is obtained from this initial interpolatedimage using edge detection masks. Alternatively, an edge map can firstbe obtained from the original input image and then this edge map isinterpolated to obtain the edge map of the output image.

Secondly, the initially obtained edge map is processed 2. It is a factthat regardless of the way that the edge map is initially obtained, itis in general only an approximation and some form of post-processing isrequired to refine the edge map. This usually is carried out to ensureconnectedness so that isolated false edge locations are eliminated.

Lastly, the edge-directed interpolation 3 of the input image is carriedout using the final edge map to provide the output image. In general,because the edge detection phase is computationally intensive, a simpleinterpolation technique may be preferred. One way to achieve this is toslightly modify bilinear interpolation to make it edge-adaptive. In thistechnique, the output pixel is computed as a weighted linear combinationof its closest four neighbours where the weights are chosen to accountfor the local edge structure.

Whilst this example obtains a high resolution edge map, a low resolutionedge map may be obtained and used in the interpolation instead, thoughwith some loss of quality in the output image. A low resolution edge mapmay be obtained directly from the low resolution input image.

In this process, the accuracy of the edge information is probably thesingle most important factor in determining the overall performance of atypical edge-directed interpolation scheme. Now, in the context of videostreaming and video transmission applications, such as digitaltelevision broadcasting or video-on-demand, it is a fact that receivers(such as television receivers, TV broadcast capture cards for personalcomputers, etc.) are designed to be as simple as possible to reduce costand in order to be robust. These simple receivers have low computationalpower and are highly optimised for a small set of predeterminedprocessing tasks (such as MPEG decoding and some mild post-processing).Accordingly, at least with current technology and at a low or mediumcost, performing accurate edge detection on the receiver side is not anoption. The alternative of using a simple edge-detection method,requiring low computational complexity, to obtain low accuracy edgeinformation results in visual artefacts (such as noisy and blurrededges, ringing and the like). If noise filtering is used prior to theedge-directed interpolation when low accuracy edge information is used,the computational load increases again, which again rules out thispossibility in practice. (Of course, for high end products, for whichthe higher component cost is acceptable to the consumer, edge detectionand processing can be carried out at the receiver by using more powerfulprocessors.)

According to a first aspect of the invention, there is provided a methodof processing of a digital image, the method comprising:

obtaining edge information of an input digital image;

transmitting digital image data corresponding to the input digital imageto a receiver;

transmitting the edge information to the receiver; and,

obtaining an output digital image at the receiver by processing thedigital image data received at the receiver using the edge informationreceived at the receiver.

By using edge information at the receiver where the edge information isprovided to the receiver and does not have to be obtained at thereceiver, the computational complexity required of the receiver is low.This means that edge-adaptive processing of the digital image data canbe carried out at the receiver in real time using a relatively low costprocessor at the receiver. This means that real-time edge-adaptivescaling of video signals can be carried out at the receiver to achievehigh quality images. As well as being used for scaling of low resolutionto higher resolution images, the invention can be used in otherprocessing techniques at the receiver, including for examplespatially-adaptive down-sampling of a high resolution image to a lowerresolution (for example, to down-sample a high definition televisionimage to a standard definition television image), conversion from oneimage standard to another, etc.

The image may be a still image but in general will be a video image,i.e. one image of many in a video stream. The image may be a frame of avideo stream and, indeed, the process may be applied to all frames of avideo stream.

The method may comprise encoding input digital image data correspondingto the input digital image prior to transmission; wherein the digitalimage data that is transmitted is the encoded input digital image data;and, wherein the processing of the digital image data received at thereceiver comprises decoding the encoded input digital image datareceived at the receiver. The encoding may comprise compressing theinput digital image data and the decoding comprises decompressing thecompressed input digital image data. In general, compression of an inputimage, such as is used in may digital image transmission protocols,leads to compression artefacts, such as blocking and ringing. Suchartefacts are usually of a high frequency nature and are recognised asedges by sub-pixel edge detection algorithms. By performing the edgedetection at for example the transmitter before the video stream iscompressed and sending the edge information separately, this problem isavoided. Some compression standards, such as MPEG, use anchor frames orso-called I-frames, and the method may be applied to such I-frames only.

In an embodiment, the encoding comprises encoding the input digitalimage data as blocks of pixels, and the obtaining of the edgeinformation comprises obtaining respective edge information for at leastsome of the blocks, and the processing of the digital image datareceived at the receiver comprises processing the pixels in said blocksrespectively according to the edge information of each of said blocks.

The processing of the digital image data received at the receiver maycomprise scaling the input digital image so that the resolution of theoutput digital image is different from the resolution of the inputdigital image.

The scaling may comprise interpolating the input digital image so thatthe resolution of the output digital image is higher than the resolutionof the input digital image, the interpolating being edge-adaptive andusing the edge information received at the receiver. In general, anyedge-adaptive interpolation method may be used.

The scaling may comprise down-sampling the input digital image so thatthe resolution of the output digital image is lower than the resolutionof the input digital image, the down-sampling being edge-adaptive andusing the edge information received at the receiver. Again, in general,any edge-adaptive down-sampling method may be used.

The processing may comprise converting the digital image data receivedat the receiver from one standard format to another standard formatusing the edge information received at the receiver. For example, thedigital image data may be converted from 720 progressive scan format to1080 progressive scan format, or from standard definition to highdefinition, etc., etc.

According to a second aspect of the invention, there is provided asystem for processing of a digital image, the system comprising:

a transmitter-side apparatus and a receiver-side apparatus;

the transmitter-side apparatus being constructed and arranged to obtainedge information of an input digital image, to transmit digital imagedata corresponding to the input digital image to the receiver-sideapparatus, and to transmit the edge information to the receiver-sideapparatus; and,

the receiver-side apparatus being constructed and arranged to obtain anoutput digital image at the receiver-side apparatus by processing thedigital image data received at the receiver-side apparatus using theedge information received at the receiver-side apparatus.

The computational burden of obtaining the edge information is placed atthe transmitter-side apparatus, where, in general, high computing poweris already available, whereas the computational requirements on thereceiver-side can be kept low.

According to a third aspect of the invention, there is provided a methodof processing of a digital image, the method comprising:

obtaining edge information of an input digital image;

transmitting digital image data corresponding to the input digital imageto a receiver; and,

transmitting the edge information to the receiver; and

whereby an output digital image can be obtained at the receiver byprocessing the digital image data received at the receiver using theedge information received at the receiver.

According to a fourth aspect of the invention, there is providedtransmitter-side apparatus for processing of a digital image, theapparatus being constructed and arranged to obtain edge information ofan input digital image, to transmit digital image data corresponding tothe input digital image to a receiver, and to transmit the edgeinformation to the receiver:

whereby an output digital image can be obtained at the receiver byprocessing the digital image data received at the receiver using theedge information received at the receiver.

According to a fifth aspect of the invention, there is provided a methodof processing of a digital image, the method comprising:

receiving at a receiver edge information of an input digital image;

receiving at the receiver digital image data corresponding to the inputdigital image; and,

obtaining an output digital image at the receiver by processing thedigital image data received at the receiver using the edge informationreceived at the receiver.

According to a sixth aspect of the invention, there is providedreceiver-side apparatus for processing of a digital image, the apparatusbeing constructed and arranged to receive transmitted edge informationof an input digital image, to receive transmitted digital image datacorresponding to the input digital image, and to obtain an outputdigital image by processing the received digital image data using theedge information.

The methods described herein may be carried out by appropriate softwarerunning on appropriate computer equipment. The software may be embeddedin an integrated circuit, the integrated circuit being adapted forperforming, or for use in the performance of, the relevant processes.Many of the processing steps may be carried out using software,dedicated hardware (such as ASICs), or a combination.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described by way of examplewith reference to the accompanying drawings, in which:

FIG. 1 shows schematically a prior art technique for edge-adaptiveinterpolation; and,

FIG. 2 shows schematically an example of an embodiment of the invention.

Referring to FIG. 2, in an example of one embodiment of the invention,edge information is extracted from an image. The edge information isconveniently and preferably in the form of a high accuracy sub-pixeledge map that is extracted by an extraction block 10 on thetransmitter-side. The edge map may be of for example all frames of aninput video image stream, or only selected frames (such as the I-frames)of an input video image stream. This extraction is carried out on aframe-by-frame basis, i.e. a separate edge map is obtained for eachframe under consideration.

In general, the edge maps may be obtained by any suitable method. Forexample, as described above, to obtain a high resolution edge map, whichis preferred in order to obtain the best quality output image, a simpleinterpolation technique (such as bilinear or bicubic interpolation) canbe used to obtain an initial interpolated image and then the estimatededge map is obtained from this initial interpolated image using edgedetection masks. Alternatively, an edge map can first be obtained fromthe original input image and then this edge map is interpolated toobtain the edge map of the output image. Either way, the initiallyobtained edge map is preferably processed to refine the edge map, againas described briefly above. Whilst it is preferred to use a highresolution edge map, a low resolution edge map may be obtained and usedin the interpolation instead, though with some loss of quality in theoutput image. A low resolution edge map may be obtained directly fromthe low resolution input image.

It should be noted that in the edge map extraction 10, the selection ofpixels may depend on the quality requirements and the specificedge-adaptive interpolation method that is ultimately used (as discussedbelow).

Next, in this embodiment, an encoder 11 encodes the input video signal.This may be in accordance with one of the known video coding standards,such as H.263/4 or MPEG-1/2/4. Optionally, the encoder 11 may compressthe data at the same time.

The encoded video signal is then transmitted over some channel 12. Aswill be well understood, the channel 12 could be one of many types or acombination, including wired or wireless, broadcast or narrowcast, overthe Internet, etc. In any event, a decoder 13 on the receiver-sidereceives the transmitted encoded video signal, decompresses it ifnecessary and decodes the received signal as appropriate.

In addition to the encoded video signal being transmitted to thereceiver-side, the corresponding edge information, which in this exampleis in the form of edge maps, is also transmitted as side information 14,preferably simultaneously with the corresponding frames of the videoimage signals. This side information 14 is received at thereceiver-side.

The received side information 14 is used to provide edge informationwhich is used in an edge-adaptive interpolation block 15 to aidinterpolation of the decoded video signal received from the decoder 13.In general, the interpolation method used in the interpolation block 15may be of any suitable type. Nevertheless, in order to keep down costsat the receiver-side (which will often be consumer equipment), atechnique requiring low computational complexity is preferred. Moreover,by labelling uniform regions where standard linear interpolation filters(such as bilinear and bicubic filters) perform satisfactorily,edge-adaptive interpolation need only be performed around high-frequencyimage details, resulting in reduced computational complexity on thereceiver-side whilst nevertheless obtaining a substantial increase invisual quality.

In the above described example, the edge information is obtained byperforming edge detection on pixels in the input image. Video codingstandards such as H.264 and MPEG are based on block processing. In thecase that block-type encoding is used, instead of performing edgedetection on pixels, the dominant edge structure in individual blockscan be detected at the transmitter-side after the input image has beenencoded. Then, at the receiver-side, the pixels within a block can beprocessed based on the edge label of the block, which is transmitted asthe side information 14. As a further enhancement, for texture blocksand blocks with no dominant edge structure, the interpolation may besimple bilinear/bicubic or other non-edge-adaptive interpolation, whichdo not make any assumptions about the edge structure. This helps toavoid producing artefacts caused by faulty edge detection and/orclassification.

In summary, embodiments enable high resolution images to be obtained byreal-time interpolation at the receiver side using edge-adaptiveinterpolation techniques. At the same time, the computationalcomplexity, and therefore the cost, required at the receiver-side is lowbecause the edge information required for the interpolation is obtainedat the transmitter side and transmitted separately as side informationto the receiver-side. This makes real-time edge-adaptive interpolationat the receiver side practical at relatively low cost.

As mentioned above, there are a number of techniques other thaninterpolation or upscaling that benefit from being edge-adaptive, suchas down-sampling (e.g. from a high definition image to a standarddefinition image), standards conversion for digital television receiversand wireless hand-held devices with video streaming capabilities, etc.The processing that is carried out at the receiver-side using separatelytransmitted edge information can be one or more of these types.

Embodiments of the invention have been described with particularreference to the examples illustrated. However, it will be appreciatedthat variations and modifications may be made to the examples describedwithin the scope of the invention.

1. A method of processing of a digital image, the method comprising:obtaining edge information of an input digital image; encoding inputdigital image data corresponding to the input digital image as blocks ofpixels; transmitting the encoded digital image data corresponding to theinput digital image to a receiver; transmitting the edge information tothe receiver; and, obtaining an output digital image at the receiver byprocessing the digital image data received at the receiver using theedge information received at the receiver, wherein the obtaining of theedge information comprises: obtaining respective edge information for atleast some of the blocks by detecting whether each individual block hasa dominant edge structure; and assigning an edge label to saidrespective block based upon said detection; and said processingcomprises: decoding the encoded input digital image data received at thereceiver; and processing the pixels in said blocks respectivelyaccording to the edge labels of each of said blocks.
 2. A methodaccording to claim 1, wherein the encoding comprises compressing theinput digital image data and the decoding comprises decompressing thecompressed input digital image data.
 3. A method according to claim 1,wherein the processing of the digital image data received at thereceiver comprises scaling the input digital image so that theresolution of the output digital image is different from the resolutionof the input digital image.
 4. A method according to claim 3, whereinthe scaling comprises interpolating the input digital image so that theresolution of the output digital image is higher than the resolution ofthe input digital image, the interpolating being edge-adaptive and usingthe edge information received at the receiver.
 5. A method according toclaim 3, wherein the scaling comprises down-sampling the input digitalimage so that the resolution of the output digital image is lower thanthe resolution of the input digital image, the down-sampling beingedge-adaptive and using the edge information received at the receiver.6. A method according to claim 1, wherein the processing comprisesconverting the digital image data received at the receiver from onestandard format to another standard format using the edge informationreceived at the receiver.
 7. A system for processing of a digital image,the system comprising: a transmitter-side apparatus and a receiver-sideapparatus; the transmitter-side apparatus being constructed and arrangedto encode input digital image data corresponding to an input digitalimage as blocks of pixels, to obtain edge information for at least someof the blocks of said input digital image by detecting whether eachindividual block has a dominant edge structure and by assigning an edgelabel to each respective block based upon said detection, to transmitthe encoded digital image data corresponding to the input digital imageto the receiver-side apparatus, and to transmit the edge information tothe receiver-side apparatus; and, the receiver-side apparatus beingconstructed and arranged to obtain an output digital image at thereceiver-side apparatus by processing the digital image data received atthe receiver-side apparatus using the edge information received at thereceiver-side apparatus, said processing comprising decoding the encodedinput digital image data received at the receiver and processing thepixels in said blocks respectively according to the edge labels of eachof said blocks.
 8. A system according to claim 7, wherein the encoder isconstructed and arranged to compress the input digital image data andthe decoder is constructed and arranged to decompress the compressedinput digital image data.
 9. A system according to claim 7, wherein thereceiver-side apparatus is constructed and arranged such that theprocessing of the digital image data received at the receiver-sideapparatus comprises scaling the input digital image so that theresolution of the output digital image is different from the resolutionof the input digital image.
 10. A system according to claim 9, whereinthe receiver-side apparatus is constructed and arranged such that thescaling comprises interpolating the input digital image so that theresolution of the output digital image is higher than the resolution ofthe input digital image, the interpolating being edge-adaptive and usingthe edge information received at the receiver-side apparatus.
 11. Asystem according to claim 9, wherein the receiver-side apparatus isconstructed and arranged such that the scaling comprises down-samplingthe input digital image so that the resolution of the output digitalimage is lower than the resolution of the input digital image, thedown-sampling being edge-adaptive and using the edge informationreceived at the receiver-side apparatus.
 12. A system according to claim7, wherein the receiver-side apparatus is constructed and arranged suchthat the processing comprises converting the digital image data receivedat the receiver-side apparatus from one standard format to anotherstandard format using the edge information received at the receiver-sideapparatus.
 13. A method of processing of a digital image, the methodcomprising: encoding input digital image data corresponding to an inputdigital image as blocks of pixels; obtaining edge information for atleast some of the blocks of said input digital image by detectingwhether each individual block has a dominant edge structure andassigning an edge label to each respective block based upon saiddetection; transmitting the encoded digital image data corresponding tothe input digital image to a receiver; and, transmitting the edgeinformation to the receiver; whereby an output digital image can beobtained at the receiver by processing the digital image data receivedat the receiver using the edge information received at the receiver,wherein said processing comprises: decoding the encoded input digitalimage data received at the receiver; and processing the pixels in saidblocks respectively according to the edge labels of each of said blocks.14. A method according to claim 13, wherein the encoding comprisescompressing the input digital image data.
 15. Transmitter-side apparatusfor processing of a digital image, the apparatus being constructed andarranged to encode input digital image data corresponding to an inputdigital image as blocks of pixels; to obtain edge information for atleast some of the blocks of said input digital image by detectingwhether each individual block has a dominant edge structure and byassigning an edge label to each respective block based upon saiddetection; to transmit the encoded digital image data corresponding tothe input digital image to a receiver, and to transmit the edgeinformation to the receiver; whereby an output digital image can beobtained at the receiver by processing the digital image data receivedat the receiver using the edge information received at the receiver,wherein the processing of the digital image data by the receivercomprises: decoding the encoded input digital image data received at thereceiver; and processing the pixels in said blocks respectivelyaccording to the edge labels of each of said blocks.
 16. Apparatusaccording to claim 15, wherein the encoded input digital image data iscompressed input digital image data.
 17. A method of processing of adigital image, the method comprising: receiving at a receiver edgeinformation of an input digital image; receiving at the receiver encodeddigital image data comprising blocks of pixels corresponding to theinput digital image; and, obtaining an output digital image at thereceiver by processing the digital image data received at the receiverusing the edge information received at the receiver, wherein the edgeinformation comprises an assigned edge label for at least some of theblocks, each edge label representing the dominant edge structure foreach respective individual block; and said processing comprises:decoding the encoded input digital image data received at the receiver;and processing the pixels in said blocks respectively according to theedge labels of each of said blocks.
 18. A method according to claim 17,wherein the encoded input digital image data is compressed input digitalimage data, the decoding comprising decompressing the compressed inputdigital image data.
 19. A method according to claim 17, wherein theprocessing of the digital image data received at the receiver comprisesscaling the input digital image so that the resolution of the outputdigital image is different from the resolution of the input digitalimage.
 20. A method according to claim 19, wherein the scaling comprisesinterpolating the input digital image so that the resolution of theoutput digital image is higher than the resolution of the input digitalimage, the interpolating being edge-adaptive and using the edgeinformation received at the receiver.
 21. A method according to claim19, wherein the scaling comprises down-sampling the input digital imageso that the resolution of the output digital image is lower than theresolution of the input digital image, the down-sampling beingedge-adaptive and using the edge information received at the receiver.22. A method according to claim 17, wherein the processing comprisesconverting the digital image data received at the receiver from onestandard format to another standard format using the edge informationreceived at the receiver.
 23. Receiver-side apparatus for processing ofa digital image, the apparatus being constructed and arranged to receivetransmitted edge information of an input digital image, to receivetransmitted digital image data encoded as blocks of pixels correspondingto the input digital image, and to obtain an output digital image byprocessing the received digital image data using the edge information,the apparatus comprising a decoder constructed and arranged to decodethe encoded input digital image data received at the receiver-sideapparatus, wherein the edge information comprises an assigned edge labelfor at least some of the blocks, each edge label representing thedominant edge structure for each respective individual block, and saidprocessing comprising processing the pixels in said blocks respectivelyaccording to the edge labels of each of said blocks.
 24. Apparatusaccording to claim 23, wherein the decoder is constructed and arrangedto decompress compressed input digital image data.
 25. Apparatusaccording to claim 23, constructed and arranged such that the processingof the digital image data received at the receiver-side apparatuscomprises scaling the input digital image so that the resolution of theoutput digital image is different from the resolution of the inputdigital image.
 26. Apparatus according to claim 25, constructed andarranged such that the scaling comprises interpolating the input digitalimage so that the resolution of the output digital image is higher thanthe resolution of the input digital image, the interpolating beingedge-adaptive and using the edge information received at thereceiver-side apparatus.
 27. Apparatus according to claim 25,constructed and arranged such that the scaling comprises down-samplingthe input digital image so that the resolution of the output digitalimage is lower than the resolution of the input digital image, thedown-sampling being edge-adaptive and using the edge informationreceived at the receiver-side apparatus.
 28. Apparatus according toclaim 23, constructed and arranged such that the processing comprisesconverting the digital image data received at the receiver-sideapparatus from one standard format to another standard format using theedge information received at the receiver-side apparatus.